🌱 fix TestClusterExtensionInstallReResolvesWhenNewCataloge2e test #1008

yashoza19 · 2024-07-02T15:52:39Z

Description

Created a new Catalog v2 with Prometheus bundle 2.0.0
Changes made to build-push-e2e-catalog.sh to allow building and pushing new catalog image
Changes made to TestClusterExtensionInstallReResolvesWhenNewCatalog e2e test -

Install ClusterExtension with existing Catalog(v1) which has Prometheus v1.2.0
Once Prometheus v1.2.0 is installed, patch the ExtensionCatalog with the newly created catalog v2
Wait and confirm that ClusterExtension picks the prometheus v2.2.0 from test-catalog:v2 and resolves to a successful install

fixes #945

Reviewer Checklist

API Go Documentation
Tests: Unit Tests (and E2E Tests, if appropriate)
Comprehensive Commit Messages
Links to related GitHub Issue(s)

netlify · 2024-07-02T15:52:55Z

✅ Deploy Preview for olmv1 ready!

Name	Link
🔨 Latest commit	`20194e8`
🔍 Latest deploy log	https://app.netlify.com/sites/olmv1/deploys/6691a490fff7c50009931dec
😎 Deploy Preview	https://deploy-preview-1008--olmv1.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

codecov · 2024-07-02T16:15:16Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 73.46%. Comparing base (60d26ed) to head (20194e8).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1008   +/-   ##
=======================================
  Coverage   73.46%   73.46%           
=======================================
  Files          32       32           
  Lines        1986     1986           
=======================================
  Hits         1459     1459           
  Misses        368      368           
  Partials      159      159

Flag	Coverage Δ
e2e	`55.89% <ø> (+0.05%)`	⬆️
unit	`45.01% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

test/e2e/e2e_suite_test.go

test/tools/build-push-e2e-catalog.sh

joelanford · 2024-07-03T20:57:03Z

test/e2e/e2e_suite_test.go

+func patchTestCatalog(ctx context.Context, name string, newImageRef string) (*catalogd.ClusterCatalog, error) {
+	// Fetch the existing ClusterCatalog
+	catalog := &catalogd.ClusterCatalog{}
+	err := c.Get(ctx, client.ObjectKey{Name: name}, catalog)
+	if err != nil {
+		return nil, err
+	}
+
+	// Update the ImageRef
+	catalog.Spec.Source.Image.Ref = newImageRef
+
+	// Patch the ClusterCatalog
+	err = c.Update(ctx, catalog)
+	if err != nil {
+		return nil, err
+	}
+
+	return catalog, err
+}


I think this is a valid test case (where we patch the spec of a ClusterCatalog), but it does not cover the case I had in mind (and a case I think we need to cover).

What I have in mind here specifically is that we need to prove that the polling behavior alone is enough to trigger an upgrade of a ClusterExtension. Forcing a change by changing the image in the ClusterCatalog spec is a side-effect foils the true intention of the test.

This "polling triggers an upgrade" test is one of the most important features of OLM, so we unfortunately can't "fake" it by touching the spec.

That was my first approach as well. Unfortunately, I can't find a way to change the latest image tag on a local registry. crane does not allow to change tags on local registries, hence I took this approach. Let me know if you you think this is still a valid test or we can approach this issue with a different solution. @joelanford

This "polling triggers an upgrade" test is one of the most important features of OLM, so we unfortunately can't "fake" it my touching the spec.

How is this accomplished in the current version of OLM? The use of make/sh/kinako(spelling), to get things into the cluster, the go to run tests, IMO blurs what is/isn't being tested, there aren't clear separation of duties in this model.

Correct me if I'm wrong, but I think you want an image loaded into the running cluster after the test has executed? Is that correct? Or do you just want this test to run longer to see if the polling picks up the the image already in the registry?

How is this accomplished in the current version of OLM?

I'm not sure

The use of make/sh/kinako(spelling), to get things into the cluster, the go to run tests, IMO blurs what is/isn't being tested, there aren't clear separation of duties in this model.

I agree that the number of moving parts is a problem. I'm fairly confident that if we wanted something a bit leaner, we could build all of this in Go.

Code in go that runs during an e2e that deploys an image registry in the cluster.

Client-go code that runs kubectl port-forward to give the e2e test a way to communicate with the registry.

go-containerregistry code to push images, move tags, etc.

Correct me if I'm wrong, but I think you want an image loaded into the running cluster after the test has executed? Is that correct?

No, I don't think that is sufficient because the polling behavior of catalogd queries a remote registry. kind load would not make it visible to the polling process. What I'm after is $something that causes the digest of a tag to change mid way through the test. That could be one of:

Pre-populating the registry with two catalog images as part of the setup, and then during the test re-tagging the second image over the first.

Pushing the first image at repo:tag as part of the setup, and then pushing the second image at that same repo:tag during the test.

joelanford · 2024-07-09T02:28:23Z

test/e2e/cluster_extension_install_test.go

+	// Tag the image with the new tag
+	baseImage := "localhost:30000/e2e/test-catalog:v1"
+	var err error
+	err = crane.Tag(baseImage, "latest", crane.Insecure)


Should "latest" be parsed from the value of testLatestCatalogRefEnvVar (instead of being hardcoded)?

testLatestCatalogRefEnvVar consist of the complete image path with latest tag. We just need to tag name and not the complete image name here

Right, but if we change that environment variable value to use a different tag, we'd have to change all the hardcoded "latest" references in the e2e code as well. If we parse the tag out of the reference, there's just the one source of truth (but also see my other comment about possibly refactoring the envvars).

joelanford · 2024-07-09T02:39:41Z

test/e2e/cluster_extension_install_test.go

+	t.Log("It resolves again when a new catalog is available")
+
+	// Tag the image with the new tag
+	baseImage := "localhost:30000/e2e/test-catalog:v1"


Is it just me or does it feel like there was some envvar sprawl before this PR, and this PR is making it more apparent with the addition of the second catalog image?

It somewhat feels like we should either:

Completely get rid of the envvars and just hardcode the references in the e2e test code

Take a more holistic look at the envvars used to carry image reference and take an opportunity to refactor/reduce

For example, should we have LOCAL_REGISTRY_HOST and CLUSTER_REGISTRY_HOST, and then every single repo/image/tag we'd use would get its own envvar with a consistent naming convention?

joelanford · 2024-07-09T02:43:22Z

test/e2e/cluster_extension_install_test.go

+	}, pollDuration, pollInterval)
+
+	// update tag on test-catalog image with v2 image
+	t.Log("By creating an ClusterExtension catalog with the updated ClusterCatalog")


This log message seems incorrecT?

Suggested change

t.Log("By creating an ClusterExtension catalog with the updated ClusterCatalog")

t.Log("By updating the catalog tag to point to the v2 catalog")

joelanford · 2024-07-09T11:49:55Z

Makefile

+export LOCAL_REGISTRY_HOST := $(E2E_REGISTRY_NAME).$(E2E_REGISTRY_NAMESPACE).svc:5000
+export CLUSTER_REGISTRY_HOST := localhost:30000


Not sure what others think, but to me it is more logical to flip these. LOCAL for localhost, CLUSTER for the service-based name.

I wouldn't call localhost a host, this is the ENDPOINT into the the cluster. It's really the external callable address. Spitballing, because names are hard, but maybe CLUSTER_REGISTRY_INGRESS?

I agree the svc:5000 address should be called CLUSTER.

joelanford · 2024-07-09T11:57:10Z

Makefile

+export V1_IMAGE_TAG := v1
+export V2_IMAGE_TAG := v2
+export LATEST_IMAGE_TAG := latest
+export CATALOG_IMG := $(LOCAL_REGISTRY_HOST)/e2e/test-catalog:$(V1_IMAGE_TAG)
+export UPDATED_CATALOG_IMG := $(LOCAL_REGISTRY_HOST)/e2e/test-catalog:$(V2_IMAGE_TAG)
+export LATEST_CATALOG_IMG := $(LOCAL_REGISTRY_HOST)/e2e/test-catalog:$(LATEST_IMAGE_TAG)
+export CLUSTER_V1_CATALOG_IMG := $(CLUSTER_REGISTRY_HOST)/e2e/test-catalog:$(V1_IMAGE_TAG)
+export CLUSTER_V2_CATALOG_IMG := $(CLUSTER_REGISTRY_HOST)/e2e/test-catalog:$(V2_IMAGE_TAG)


Two comments here:

I don't think we need envvars for every combo. Let's just concatenate.

Let's pick a convention like <repoName>_<imageName_<tag> and use that.

I don't think we'll need a "latest" variant here anymore. The test that needs it can deal with pushing its own tag local to the test.

So I think all of this can boil down to:

Suggested change

export V1_IMAGE_TAG := v1

export V2_IMAGE_TAG := v2

export LATEST_IMAGE_TAG := latest

export CATALOG_IMG := $(LOCAL_REGISTRY_HOST)/e2e/test-catalog:$(V1_IMAGE_TAG)

export UPDATED_CATALOG_IMG := $(LOCAL_REGISTRY_HOST)/e2e/test-catalog:$(V2_IMAGE_TAG)

export LATEST_CATALOG_IMG := $(LOCAL_REGISTRY_HOST)/e2e/test-catalog:$(LATEST_IMAGE_TAG)

export CLUSTER_V1_CATALOG_IMG := $(CLUSTER_REGISTRY_HOST)/e2e/test-catalog:$(V1_IMAGE_TAG)

export CLUSTER_V2_CATALOG_IMG := $(CLUSTER_REGISTRY_HOST)/e2e/test-catalog:$(V2_IMAGE_TAG)

export E2E_TESTCATALOG_V1 := e2e/testcatalog:v1

export E2E_TESTCATALOG_V2 := e2e/testcatalog:v2

And then when we need to reference a fully-qualified image, we would just use, for example:

$(LOCAL_REGISTRY_HOST)/$(E2E_TEST_CATALOG_V1)

but we still need to export the cluster image as CLUSTER_V1_CATALOG_IMG := $(CLUSTER_REGISTRY_HOST)/$(E2E_TEST_CATALOG_V1) in the makefile, right?

Correct me if I'm wrong @joelanford, but I think you want less stuff in the Makefile, and more of the work in the tests that need "additional" things. IE instead of exported vars, concatenate them in the go code? So it would be something like:

fmt.Sprintf("%s/%s", os.Getenv("LOCAL_REGISTRY_HOST"), os.Getenv("E2E_TEST_CATALOG_V1"))

Or did you mean something entirely different.

What @acornett21 said. I think there may be at least one place in the Makefile where we have to build/push the image, and that needs the fully-qualifed image reference.

In that case, I'd concatenate them just right at the call site rather than making an extra variable to store the concatenation.

This has been implemented now in the latest commit. Thanks

joelanford · 2024-07-11T00:52:38Z

Makefile

+export CLUSTER_REGISTRY_HOST := localhost:30000
+export E2E_TEST_CATALOG_V1 := e2e/test-catalog:v1
+export E2E_TEST_CATALOG_V2 := e2e/test-catalog:v2
+export CATALOG_IMG := $(LOCAL_REGISTRY_HOST)/$(E2E_TEST_CATALOG_V1)


Do we need CATALOG_IMG variable anymore?

Yes, it is required here: https://github.com/operator-framework/operator-controller/blob/main/test/extension-developer-e2e/setup.sh#L33

joelanford · 2024-07-12T21:15:53Z

test/e2e/e2e_suite_test.go

@@ -24,6 +25,7 @@ var (
 const (
 	testCatalogRefEnvVar = "CATALOG_IMG"
 	testCatalogName      = "test-catalog"
+	LatestImageTag       = "latest"


Is there a reason to export this? I'd suggest unexporting:

Suggested change

LatestImageTag = "latest"

latestImageTag = "latest"

I'll hit this in a quick follow-up.

I've made the changes in this PR itself

Signed-off-by: yashoza19 <yoza@redhat.com>

yashoza19 requested a review from a team as a code owner July 2, 2024 15:52

acornett21 reviewed Jul 2, 2024

View reviewed changes

test/e2e/e2e_suite_test.go Outdated Show resolved Hide resolved

yashoza19 force-pushed the issue945 branch 2 times, most recently from 09ff7f3 to 4065eb1 Compare July 2, 2024 22:45

joelanford reviewed Jul 3, 2024

View reviewed changes

test/tools/build-push-e2e-catalog.sh Outdated Show resolved Hide resolved

joelanford reviewed Jul 3, 2024

View reviewed changes

yashoza19 force-pushed the issue945 branch 3 times, most recently from 8df658c to 15642b7 Compare July 9, 2024 02:16

joelanford reviewed Jul 9, 2024

View reviewed changes

yashoza19 force-pushed the issue945 branch 3 times, most recently from 75b2968 to 8ea854c Compare July 9, 2024 04:22

yashoza19 requested a review from joelanford July 9, 2024 04:33

joelanford reviewed Jul 9, 2024

View reviewed changes

yashoza19 force-pushed the issue945 branch from 8ea854c to c9ffa23 Compare July 9, 2024 15:35

openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 9, 2024

yashoza19 force-pushed the issue945 branch from c9ffa23 to c0e9639 Compare July 9, 2024 15:51

openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 9, 2024

yashoza19 force-pushed the issue945 branch 2 times, most recently from b805ce9 to a9f83ce Compare July 9, 2024 17:13

yashoza19 requested a review from joelanford July 10, 2024 19:02

joelanford reviewed Jul 11, 2024

View reviewed changes

yashoza19 requested a review from joelanford July 12, 2024 17:47

joelanford reviewed Jul 12, 2024

View reviewed changes

fix TestClusterExtensionInstallReResolvesWhenNewCataloge2e test

20194e8

Signed-off-by: yashoza19 <yoza@redhat.com>

yashoza19 force-pushed the issue945 branch from a9f83ce to 20194e8 Compare July 12, 2024 21:47

yashoza19 requested a review from acornett21 July 15, 2024 16:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🌱 fix TestClusterExtensionInstallReResolvesWhenNewCataloge2e test #1008

🌱 fix TestClusterExtensionInstallReResolvesWhenNewCataloge2e test #1008

yashoza19 commented Jul 2, 2024 •

edited

Loading

netlify bot commented Jul 2, 2024 •

edited

Loading

codecov bot commented Jul 2, 2024 •

edited

Loading

joelanford Jul 3, 2024 •

edited

Loading

yashoza19 Jul 3, 2024

acornett21 Jul 3, 2024

joelanford Jul 4, 2024

joelanford Jul 9, 2024

yashoza19 Jul 9, 2024

joelanford Jul 9, 2024

joelanford Jul 9, 2024 •

edited

Loading

joelanford Jul 9, 2024 •

edited

Loading

joelanford Jul 9, 2024

acornett21 Jul 9, 2024

joelanford Jul 9, 2024 •

edited

Loading

yashoza19 Jul 9, 2024

acornett21 Jul 9, 2024

joelanford Jul 9, 2024

yashoza19 Jul 9, 2024

joelanford Jul 11, 2024

yashoza19 Jul 11, 2024

joelanford Jul 12, 2024

joelanford Jul 12, 2024

yashoza19 Jul 12, 2024

	t.Log("By creating an ClusterExtension catalog with the updated ClusterCatalog")
	t.Log("By updating the catalog tag to point to the v2 catalog")

		export LOCAL_REGISTRY_HOST := $(E2E_REGISTRY_NAME).$(E2E_REGISTRY_NAMESPACE).svc:5000
		export CLUSTER_REGISTRY_HOST := localhost:30000

🌱 fix TestClusterExtensionInstallReResolvesWhenNewCataloge2e test #1008

Are you sure you want to change the base?

🌱 fix TestClusterExtensionInstallReResolvesWhenNewCataloge2e test #1008

Conversation

yashoza19 commented Jul 2, 2024 • edited Loading

Description

Reviewer Checklist

netlify bot commented Jul 2, 2024 • edited Loading

✅ Deploy Preview for olmv1 ready!

codecov bot commented Jul 2, 2024 • edited Loading

Codecov Report

joelanford Jul 3, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joelanford Jul 9, 2024 • edited Loading

Choose a reason for hiding this comment

joelanford Jul 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joelanford Jul 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yashoza19 commented Jul 2, 2024 •

edited

Loading

netlify bot commented Jul 2, 2024 •

edited

Loading

codecov bot commented Jul 2, 2024 •

edited

Loading

joelanford Jul 3, 2024 •

edited

Loading

joelanford Jul 9, 2024 •

edited

Loading

joelanford Jul 9, 2024 •

edited

Loading

joelanford Jul 9, 2024 •

edited

Loading